Skip to content

feat(web): add web generator #285

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat(web): add web generator #285

wants to merge 2 commits into from

Conversation

avivkeller
Copy link
Member

@avivkeller avivkeller commented May 28, 2025

Fixes #7.

This PR adds the web generator.

Tasks / Issues

P1 – Must Complete Before Merge

  • Improve generator performance (PostCSS + Minification)
  • Optimize client-side bundle size (Waiting on Orama)
  • Add more items (anyone can do this as they review1)

P2 – Must complete before migration

  • ChangeHistory component doesn’t render Markdown correctly. (This an issue with JSX generator, not the web generator)
  • Add more items (anyone can do this as they review1)

P3 – Can Be Done in a Follow-up

  • Add more items (anyone can do this as they review1)

Get a preview

node bin/cli.mjs generate -t orama-db -i "/node/doc/api/*.md" -o "./out" # If you want search functionality
node bin/cli.mjs generate -t web -i "/node/doc/api/*.md" -o "./out" --index "/node/doc/api/index.md" # Specifying `--index` is optional, but recommended
npx serve out # You can serve the output, or just open one of the files in your browser. Serving is required for using search.

Footnotes

  1. Add things as they appear, or leave review comments. 2 3

@codecov-commenter
Copy link

codecov-commenter commented May 28, 2025

Codecov Report

Attention: Patch coverage is 86.88969% with 145 lines in your changes missing coverage. Please review.

Project coverage is 63.30%. Comparing base (b3e2fbd) to head (dfae938).

✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/generators/jsx-ast/index.mjs 0.00% 38 Missing ⚠️
src/generators/web/index.mjs 79.03% 26 Missing ⚠️
src/generators/jsx-ast/utils/buildContent.mjs 54.71% 24 Missing ⚠️
src/generators/orama-db/index.mjs 57.14% 18 Missing ⚠️
src/generators/web/build/bundle.mjs 91.02% 7 Missing ⚠️
src/generators/jsx-ast/utils/buildBarProps.mjs 50.00% 5 Missing and 1 partial ⚠️
src/generators/index.mjs 0.00% 5 Missing ⚠️
src/generators/web/build/plugins.mjs 95.18% 4 Missing ⚠️
src/utils/remark.mjs 50.00% 4 Missing ⚠️
...rc/generators/legacy-html/utils/buildDropdowns.mjs 0.00% 3 Missing ⚠️
... and 7 more
Additional details and impacted files
@@             Coverage Diff             @@
##             main     #285       +/-   ##
===========================================
+ Coverage   51.15%   63.30%   +12.15%     
===========================================
  Files          81      128       +47     
  Lines        6956    10471     +3515     
  Branches      306      613      +307     
===========================================
+ Hits         3558     6629     +3071     
- Misses       3395     3838      +443     
- Partials        3        4        +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@avivkeller avivkeller marked this pull request as ready for review May 28, 2025 22:13
@avivkeller avivkeller requested a review from a team as a code owner May 28, 2025 22:13
@avivkeller avivkeller marked this pull request as draft May 29, 2025 16:36
const language = matches?.groups?.language ?? '';
const [copyText, setCopyText] = useState('Copy to clipboard');

const handleCopy = async text => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not including useCopyToClipord in ui lib ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC I didn't include it since it didn't fall under the "UI Component" category, it's a hook, however, if you disagree, I won't block.

"estree-util-value-to-estree": "^3.4.0",
"estree-util-visit": "^2.0.0",
"github-slugger": "^2.0.0",
"glob": "^11.0.2",
"hast-util-to-string": "^3.0.1",
"hastscript": "^9.0.1",
"html-minifier-terser": "^7.2.0",
"mustache": "^4.2.0",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, why?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to use __<VARIABLE>__ like in the other generators, however, the JavaScript I was trying to add to the HTML broke the structure of the file. Using a library like Mustache properly escapes anything injected.

Copy link
Member

@AugustinMauroy AugustinMauroy May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

export const TEMPLATE = ({...props}) => dedent`
<html>
 ${props.foo}
</html>
`; 

this can work

Copy link
Member Author

@avivkeller avivkeller May 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry if I wasn’t clear, the compiled JavaScript broke the HTML structure because it contained characters that need to be escaped to be in an HTML script.

For example, in your scenario, if props.foo contains a </html>, it’ll escape the sandbox.

Mustache performs all the needed escaping.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like that we're using yet another layer of whatever freakery here. Mustache is great, but it feels like you're throing a nuke at an ant's sized problem. Not to mention adding yet another engine here will make things even more complex and costly-performance wise. Let's try to avoid throwing npm packages to any problem we encounter, shall we?

@avivkeller
Copy link
Member Author

avivkeller commented May 30, 2025

🎉 The code now dehydrates to the client so it can render without JavaScript!

@AugustinMauroy
Copy link
Member

🎉 The code now dehydrates to the client so it run without JavaScript!

Wow 😵‍💫 and what about codetab

@avivkeller
Copy link
Member Author

avivkeller commented May 30, 2025

It rehydrates and runs with JS, but if you don't have JS, you can still view the docs. I used React's SSRing

@avivkeller
Copy link
Member Author

@AugustinMauroy and I got search to finally work 🎉
image

@avivkeller avivkeller force-pushed the feat/web/gen branch 2 times, most recently from 024bbea to dbfe55d Compare June 3, 2025 21:40
@avivkeller
Copy link
Member Author

@nodejs/nodejs-website @nodejs/web-infra ChangeHistory and SideBar aren't implemented yet, so this is still a draft, but it's ready for you to take a look, so feel free to review :-)

@avivkeller avivkeller linked an issue Jun 6, 2025 that may be closed by this pull request
@ovflowd
Copy link
Member

ovflowd commented Jun 24, 2025

Generating the docs on my machine only with the "web" generator takes an awful lot of time... Almost 2 minutes 😅

Yes, it's unfortunately not very fast at the moment, but I plan to improve this at some point before this gets to core.

I think it should be improved prior to merge.

@avivkeller
Copy link
Member Author

avivkeller commented Jun 24, 2025

@ovflowd

OOC: "The client-side bundle is large", how large

image

Around 1.3mb, with 1mb coming from Orama. The actual page content is ~96kb.

Why on nodejs.org-side? This is about rendering the Markdown into HTML, no?

The position of the label in AlertBox comes from the CSS in @node-core/ui-components.

I'm assuming Stability Index component takes the content as a "child"?

Yes. The stability index component is AlertBox, so it takes the level and title as a property, and the content as children.

Generating the docs on my machine only with the "web" generator takes an awful lot of time... Almost 2 minutes 😅
I think it should be improved prior to merge.

Looking at the Flamegraph, to generate everything (for me) takes 136s1, the "hottest" (longest) pieces were PostCSS (especially Tailwind) and minification, followed by Shiki. For reference, without minification, the process takes roughly 110s.

@Harkunwar said he was going to work on compiling the @node-core/ui-components library during pre-publishing, which would drop the PostCSS piece, since it'll already be compiled to minified css.

What's this supposed to be? 🤔

When generated without the --index flag (which has been newly added), the titles are inferred from the API metadata. index.md has no title. Once the tooling lands in core, we can open a PR to title this document, or just always pass the --index flag.

I feel this is awfully close to the sidebar

Would you prefer we hide the scrollbar, like we do on the right side?

image

If I recall we wanted to have a new component to render this nicely. I think it was never designed; But I believe we wanted something that rendered nicely the method signature.

I'm open to a new component. If you'd like, I can give you edit access to the Figma (Slack message me your Figma user-email) for designing.

Should we switch this PR back to draft and address all the concerns on this PR (thinking this PR as a feature branch); I feel I'd rather get this merged once most of the critical pendencies are resolved and the concerns raised by me and Dario.

If you'd like to, we most definitely can, but it is in a state where it's ready for review, so I wouldn't want the draft state to discourage thorough review.


@dario-piotrowicz

the left aside nav links don't have a hover effect, is that intentional?

It's the same on https://nodejs.org, so any changes on that front should be done in @node-core/ui-components.

the links on the right aside (table of contents, etc...) do have a hover effect but the difference is practically non-existent (in dark mode is #fff vs #e9edf0)

Ditto, see @node-core/ui-components.

there are some elements that have the exact same style in light mode (but they look to be styled for dark mode for me)

  • the search element on the top

Fixed in the latest commit.

  • code snippets

It's the same on https://nodejs.org. IIRC we had a light-mode design, but it's in the Figma graveyard.

I personally think the stability alter boxes should include the word stability (right now I only see something like 0 Experimental which I understand because of my familiarity with the nodejs api docs, however I don't think that this is new-comer friendly, ideally it should be, more similar to the current one where it says Stability 0: Experimental)

In our Figma design, the word "Stability" is not included, however, would you prefer one of the designs below?

image
image

claudio already mentioned the version picker, I would also like to add the the width of the element makes the left aside have a very ugly/unnecessary horizontal scrollbar on smaller (but not mobile) window widths)

As I mentioned above, "Would you prefer we hide the scrollbar, like we do on the right side?"

the Toggle navigation menu element is not very accessible, it has a role of button and an aria-label, but it doesn't seem like it is interactable via keyboard (tab navigation + enter)

That's an issue for @node-core/ui-components, it's the same behavior on https://nodejs.org.

the history selector's focus-visible style is barely noticeable in light mode

Ditto, see @node-core/ui-components.

for some reason the whole document body is focusable via keyboard (/tab) navigation

I just tab-ed through the entire page, and didn't see a point where the entire body was highlighted.

Out of curiosity I also run an accessibility audit using axe DevTools and it has found 5 serious issues

Fixed the HTML language issue in the latest commit, see nodejs/nodejs.org#7890 for the contrast issue, other issues are inherited from @node-core/ui-components, i.e. the list not containing only li.

Footnotes

  1. This is running a single-thread, since it's harder to use Flamegraphs when multiple processes (workers) are involved. However, since this is only a single generator, threading wouldn't improve speeds at all. (w/ threading: 139s, so basically the same speed)

@ovflowd
Copy link
Member

ovflowd commented Jun 24, 2025

Around 1.3mb, with 1mb coming from Orama. The actual page content is ~96kb.

Jesus christ. Can we make the bundle of orama stuff separated? With a React Lazy import?

@ovflowd
Copy link
Member

ovflowd commented Jun 24, 2025

The position of the label in AlertBox comes from the CSS in @node-core/ui-components.

I was referring to the unstyled Markdown in the content of the AlertBox

image

@Harkunwar said he was going to work on compiling the @node-core/ui-components library during pre-publishing, which would drop the PostCSS piece, since it'll already be compiled to minified css.

Nice.

I'm open to a new component. If you'd like, I can give you edit access to the Figma (Slack message me your Figma user-email) for designing.

Let's mock something outside of Figma. We can open an issue and come up with ideas tbh. It's just the current rendering is definitely not great

If you'd like to, we most definitely can, but it is in a state where it's ready for review, so I wouldn't want the draft state to discourage thorough review.

Fair. Can we add a blocked label?

Would you prefer we hide the scrollbar, like we do on the right side?

Are these the same paddings from the website? Or is it because we don't have any selector on non-mobile for navigation on website? 🤔 But I'd rather not hiding the scrollbar on the left-side. Maybe just adjusting the padding.

In our Figma design, the word "Stability" is not included, however, would you prefer one of the designs below?

Fine with either designs, but I actually prefer the pill.


Extra feedback:

  • Since API docs are built and set in stone, I want us to have a feature so that it reads remote "api-docs" config files that includes warnings/banners/pills that we can put out globally. So we are able to have the same banners (and even different ones, such as ESP,EOL) on the API Docs website.
    • The JSON map could have some global stuff (like updated logo, favicon, or etc) and also per version one:
    • { v24: { banners: {}; pills: {} } }

@avivkeller
Copy link
Member Author

I was referring to the unstyled Markdown in the content of the AlertBox

image

That has since been fixed

@avivkeller
Copy link
Member Author

Jesus christ. Can we make the bundle of orama stuff separated? With a React Lazy import?

Per Michele Riva on the progress of making Orama tree-shakable: "Yes we are actively working on it! We are testing the components before releasing them in the coming week. We’ll open a PR when they're ready"

So, let's see if they have something for us in the near future (this/next week), and if not, move forward with lazy loading.

@avivkeller
Copy link
Member Author

@ovflowd I've done a potential mock with modified <details/> tags (so no JS!), wdyt1:
image
image
image
image
image

Footnotes

  1. These are mocks, and would still need to be implemented in our ui-components, and might not be realistic. The data shown is placeholder, and doesn't actually reflect the docs

@ovflowd
Copy link
Member

ovflowd commented Jun 24, 2025

I was more thinking of what I shared before. We can reconstruct what would be the method structure:

methodName(arg1, arg1, arg3,...): ReturnType1
methodName(arg1, argh): ReturnType2

(Think the above as the method having overloads if any). These would be within a code snippet as if they were a type definition of sort (see the link to next.js docs I've attached); We do have the structure of the method which is generated by the JSON Generator (and I suppose also resides on the AST?)

And then below this we would have a table with the argument names, description, etc...

That IMO would look the cleanest.

@avivkeller
Copy link
Member Author

Ahh I didn't see the link reference, I understand now. Thanks!

@dario-piotrowicz
Copy link
Member

the left aside nav links don't have a hover effect, is that intentional?

It's the same on https://nodejs.org, so any changes on that front should be done in @node-core/ui-components.

the links on the right aside (table of contents, etc...) do have a hover effect but the difference is practically non-existent (in dark mode is #fff vs #e9edf0)

Ditto, see @node-core/ui-components.

Yeah, I see 👀 , anyways I do feel that these are not alright, but I agree that then it is out of scope here 🙂👍

there are some elements that have the exact same style in light mode (but they look to be styled for dark mode for me)

  • the search element on the top

Fixed in the latest commit.

Looks great! Thanks! 😄

  • code snippets

It's the same on https://nodejs.org. IIRC we had a light-mode design, but it's in the Figma graveyard.

ok then again out of scope but ideally something to follow up on 🙂👍

I personally think the stability alter boxes should include the word stability (right now I only see something like 0 Experimental which I understand because of my familiarity with the nodejs api docs, however I don't think that this is new-comer friendly, ideally it should be, more similar to the current one where it says Stability 0: Experimental)

In our Figma design, the word "Stability" is not included, however, would you prefer one of the designs below?

image image

The second one for sure for me! 😄

claudio already mentioned the version picker, I would also like to add the the width of the element makes the left aside have a very ugly/unnecessary horizontal scrollbar on smaller (but not mobile) window widths)

As I mentioned above, "Would you prefer we hide the scrollbar, like we do on the right side?"

I can see that you've hidden the scrollbar... no that's not the right solution in my opinion, I think that the version picker should have a max-width so that it doesn't overflow its container (causing the horizontal scrollbar not to appear at all since it won't be needed)

the Toggle navigation menu element is not very accessible, it has a role of button and an aria-label, but it doesn't seem like it is interactable via keyboard (tab navigation + enter)

That's an issue for @node-core/ui-components, it's the same behavior on https://nodejs.org.

the history selector's focus-visible style is barely noticeable in light mode

Ditto, see @node-core/ui-components.

Ok, sorry for all of these @node-core/ui-components comments here 😓

for some reason the whole document body is focusable via keyboard (/tab) navigation

I just tab-ed through the entire page, and didn't see a point where the entire body was highlighted.

Yes... I am not seeing that anymore... I 100% saw that though... I'll try to see if it happens only after certain conditions 😕

Out of curiosity I also run an accessibility audit using axe DevTools and it has found 5 serious issues

Fixed the HTML language issue in the latest commit, see nodejs/nodejs.org#7890 for the contrast issue, other issues are inherited from @node-core/ui-components, i.e. the list not containing only li.

👍

@avivkeller
Copy link
Member Author

@dario-piotrowicz for your @node-core/ui-components concerns, do you mind opening a catch all issue in nodejs.org, so we can follow up?

@dario-piotrowicz
Copy link
Member

Now there's an evident flickering on the search element (also on the theme toggle button) in dark mode when I refresh the page:

simplescreenrecorder.mp4

(PS: they flicker in light mode too but not as evidently)

@dario-piotrowicz
Copy link
Member

@dario-piotrowicz for your @node-core/ui-components concerns, do you mind opening a catch all issue in nodejs.org, so we can follow up?

do you mind if I open self isolated issues? (as they might be easier to deal with?)

or maybe an issue with sub-issues?

@avivkeller
Copy link
Member Author

do you mind if I open self isolated issues? (as they might be easier to deal with?)

or maybe an issue with sub-issues?

Same difference, just figured catch all might be less work for you, but that's even better :-)

@dario-piotrowicz
Copy link
Member

do you mind if I open self isolated issues? (as they might be easier to deal with?)
or maybe an issue with sub-issues?

Same difference, just figured catch all might be less work for you, but that's even better :-)

Ok I'll open isolated issues there 🙂👍

@avivkeller
Copy link
Member Author

avivkeller commented Jun 25, 2025

(PS: they flicker in light mode too but not as evidently)

It's actually the Orama component loading in (which moves the toggle). Once Orama fixes their components, it'll be partially statically dehydrated, filling in the space

@dario-piotrowicz
Copy link
Member

In order for search to work, you need to run:

node bin/cli.mjs generate -t orama-db -i "/node/doc/api/*.md" -o "./out"

To generate the database (orama-db.json)

Sorry for not following the PR instructions earlier 🙂

Yes I did run the orama-db generation and that does enable search for me! 😃

@avivkeller
Copy link
Member Author

And then below this we would have a table with the argument names, description, etc...

I'm just concerned what happens when there is a lot of nesting, for example,

Parameter Type Description
hostname <string> The hostname to resolve.
options <integer> | <Object> Optional flags or configuration object.
└─ family <integer> | <string> The record family: 4, 6, or 0. 'IPv4' and 'IPv6' are interpreted as 4 and 6. 0 allows either IPv4 or IPv6. With { all: true }, returns one or both address types depending on the system. Default: 0.
└─ hints <number> One or more getaddrinfo flags, bitwise ORed if multiple.
└─ all <boolean> If true, returns all resolved addresses in an array. Otherwise, returns a single address. Default: false.
└─ order <string> Controls sorting of addresses: verbatim, ipv4first, or ipv6first. Default: verbatim. Configurable via dns.setDefaultResultOrder() or --dns-result-order.
└─ verbatim <boolean> If true, returns addresses in the order provided by the DNS resolver. If false, IPv4 addresses precede IPv6. Deprecated in favor of order. If both are set, order takes precedence. Default: true.
callback <Function> Callback function for the resolution.
└─ err <Error> Error object if resolution fails.
└─ address <string> The resolved IPv4 or IPv6 address.
└─ family <integer> 4 or 6, representing the address family. 0 indicates a possible bug in the name resolution system.

That's going to be quite a table, no?

@avivkeller
Copy link
Member Author

As a personal design preference, I quite like the expandable function signatures, I think it gives a really cool way of displaying the parameters.

For events, I was thinking of it showing on("[name]", (...) => {}) rather than the current layout; this also gives users a way to see how to access events (if they don't already know)

@ovflowd
Copy link
Member

ovflowd commented Jun 25, 2025

And then below this we would have a table with the argument names, description, etc...

I'm just concerned what happens when there is a lot of nesting, for example,

Parameter Type Description
hostname <string> The hostname to resolve.
options <integer> | <Object> Optional flags or configuration object.
└─ family <integer> | <string> The record family: 4, 6, or 0. 'IPv4' and 'IPv6' are interpreted as 4 and 6. 0 allows either IPv4 or IPv6. With { all: true }, returns one or both address types depending on the system. Default: 0.
└─ hints <number> One or more getaddrinfo flags, bitwise ORed if multiple.
└─ all <boolean> If true, returns all resolved addresses in an array. Otherwise, returns a single address. Default: false.
└─ order <string> Controls sorting of addresses: verbatim, ipv4first, or ipv6first. Default: verbatim. Configurable via dns.setDefaultResultOrder() or --dns-result-order.
└─ verbatim <boolean> If true, returns addresses in the order provided by the DNS resolver. If false, IPv4 addresses precede IPv6. Deprecated in favor of order. If both are set, order takes precedence. Default: true.
callback <Function> Callback function for the resolution.
└─ err <Error> Error object if resolution fails.
└─ address <string> The resolved IPv4 or IPv6 address.
└─ family <integer> 4 or 6, representing the address family. 0 indicates a possible bug in the name resolution system.
That's going to be quite a table, no?

I mean sure, but any other design would also be quite the thing. I guess we could refine. For example if you have 2 variations we could have:

methodName(arg1, arg1, arg3,...): ReturnType1

TABLE CONTAINING VARIATION ONE

methodName(arg1, argh): ReturnType2

TABLE CONTAINING VARIATION TWO?

@ovflowd
Copy link
Member

ovflowd commented Jun 25, 2025

To be honest, I like this design:

image

But it doesn't speak our design language and it is a bit complex visually speaking. The one from the example of Next.js docs is simple and easy to visualize. With variation we can adopt many different paths and experiment.

@avivkeller
Copy link
Member Author

Okay, I'll program the table, and we'll see how it looks.

@avivkeller
Copy link
Member Author

@ovflowd (Note: This is a first draft to show what a table would look it more-or-less, i.e. Property will be changed to Parameter, etc)

image
image

@ovflowd
Copy link
Member

ovflowd commented Jun 25, 2025

Nice, I still think we need to add that snippet of method signature that I mentioned. This would allow us to rename the header section to be simply the method name or class name. It also helps on Search Indexing

@avivkeller
Copy link
Member Author

Nice, I still think we need to add that snippet of method signature that I mentioned. This would allow us to rename the header section to be simply the method name or class name. It also helps on Search Indexing

Yes, the draft was just for the table, but I plan to use parseSignature (or a variation of) for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocked enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add react-web generator Write React Components
6 participants